Skip to content

fix: split Harness name and version in persona analytics (CLI-1771) - #7162

Merged
nick-y-snyk merged 15 commits into
mainfrom
fix/CLI-1771-harness-name-version-split
Aug 21, 2026
Merged

fix: split Harness name and version in persona analytics (CLI-1771)#7162
nick-y-snyk merged 15 commits into
mainfrom
fix/CLI-1771-harness-name-version-split

Conversation

@nick-y-snyk

@nick-y-snyk nick-y-snyk commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Summary

  • persona.agent fused Claude Code's name, version, and role into one string (claude-code_2-1-233_agent), fragmenting it into ~101 distinct values in analytics and making "how many Claude Code sessions" impossible to query. This splits name and version into two fields.
  • Replaces the hand-ported detect-agent signature table (cliv2/internal/persona/agent) with the upstream github.com/vercel/detect-agent module — grows coverage 13 → 20 harnesses and fixes a live bug where plain-CLI OpenCode was undetected (our port matched the launcher's env var, not the one OpenCode itself sets).
  • Adds a harness-agnostic regex that splits a version-shaped suffix off any detected name (name[_/]version[_/role]), and canonicalises the remaining fragment against detect-agent's own KnownAgents vocabulary rather than a hand-written alias list.
  • New persona.agent_version is written only when a version is found; persona.agent always resolves to claude_code for Claude Code regardless of whether it was detected via AI_AGENT or via environment signature.
  • cursor-cli, augment-cli, and every other already-clean harness are unaffected — no version-shaped suffix, so the regex doesn't match and the value passes through unchanged.

Signature renames, flagging ahead of release per the spec: dropping the hand-ported table for upstream detect-agent's own vocabulary changes the canonical string several harnesses report. None of these are remapped back to the old values — intentional, current analytics data for these harnesses is already messed up, so this is a good time to adopt vercel's naming instead of preserving a value we hand-picked ourselves:

Detection case Old DetectAgent() New Detect()
Claude Code signature claude claude_code
Gemini signature gemini gemini_cli
Codex signature codex codex_cli
OpenCode signature opencode open_code
AI_AGENT=github-copilot-cli github-copilot github-copilot-cli

This lands in the next release; will also be called out as a Jira comment on CLI-1771.

Datadog: where the new field lands

Confirmed against live analytics-service logs (service:analytics-service "analytics payload"). Today:

custom.analytics-service.interaction.extension.persona.agent = "claude-code_2-1-220_harness"

After this change, agent_version is a new sibling key at the same nesting level, queryable as @analytics-service.interaction.extension.persona.agent_version:

persona:
    agent: claude_code
    agent_version: "2.1.220"
    interactive: false
    interactive_mode: 0

agent_version is omitted from the payload entirely (not an empty string) whenever no version is parsed out.

License bundling — fixed

github.com/vercel/detect-agent has no LICENSE file in its repo (confirmed via the GitHub API), despite tagged releases and external contributors. This hard-failed CI and any local build.

Every third-party Go dependency in cliv2 ships its license text embedded in the CLI binary: cliv2/scripts/prepare_licenses.go runs go-licenses save ./... and writes one LICENSE (or NOTICE/COPYING) file per module path under cliv2/internal/embedded/_data/licenses/, which cliv2/internal/embedded/file.go pulls into the binary via //go:embed _data and surfaces to end users as the OSS notices. This step runs as part of make configure in cliv2/Makefile, ahead of make build.

Reproduced locally on this branch, matches CI exactly:

E... library.go:122] Failed to find license for github.com/vercel/detect-agent: cannot find a known open source license for ".../github.com/vercel/detect-agent@v1.2.0" ...
F... main.go:77] one or more libraries have an incompatible/unknown license: map["unknown":["github.com/vercel/detect-agent"]]
Error running go-licenses save: exit status 1

So make configure / make build fail outright with this dependency in place — this is not a merge that could accidentally ship unnoticed; the build gate stops it. go test ./... still passes (it doesn't go through make configure), which is why the test-plan checkboxes below are green despite this blocker.

The script already has a manual-override mechanism for packages go-licenses can't auto-detect (github.com/davecgh/go-spew, github.com/alexbrainman/sspi, github.com/pmezard/go-difflib, go.dev are each fetched from a hardcoded URL) — but go-licenses save fatal-exits before reaching that step for a package it finds zero license-like files for at all, so the override alone wasn't reachable. Fixed with two changes in cliv2/scripts/prepare_licenses.go:

  1. --ignore github.com/vercel/detect-agent on the go-licenses save invocation, so it no longer requires a verdict on that package (dependencies of the ignored package would still be checked, but it has none).
  2. A new manual-download entry supplying the Apache-2.0 license text directly (https://www.apache.org/licenses/LICENSE-2.0.txt), matching github.com/vercel/detect-agent's own package.json declaration ("license": "Apache-2.0") — its repo just never committed the file.

Verified both changes are needed: removing the --ignore and keeping only the manual-download entry still reproduces the exact same fatal error, since go-licenses save never reaches the manual-download loop. With both in place, go run scripts/prepare_licenses.go exits 0 and embeds a real Apache-2.0 LICENSE file for github.com/vercel/detect-agent.

Note this asserts Apache-2.0 based on package.json's declaration, not on a license file the copyright holder actually published — still worth an explicit legal nod given that.

Test plan

  • go test ./... passes in cliv2 and cliv2-private
  • go vet ./... and golangci-lint run clean on changed packages
  • go mod tidy -diff clean in both cliv2 and cliv2-private
  • New test coverage in cliv2/internal/persona/persona_test.go (TestReport_Agent) at the single seam where persona values are recorded onto analytics — extends the existing fake-analytics-recorder test rather than a new harness, using t.Setenv per case (the upstream module reads the process environment directly, no injection point). Covers every real-world shape observed in production: fused underscore (both _agent and _harness roles), legacy slash format normalising to the same bucket, bare signature detection, an already-clean harness, a fused identifier with no version-shaped segment, an identifier that fails to canonicalise, a redacted (***) identifier, and a version-shaped run with no trailing role.
  • Verified against 14 days of production analytics-service logs: no harness other than Claude Code currently reports a version-shaped persona.agent value at any real volume, confirming the ticket's Out-of-Scope call.
  • Manual: after merge, re-run the distinct-persona.agent-value count and confirm Claude Code's ~101 buckets collapse to one, with persona.agent_version populated on those events.

Ref: CLI-1771

persona.agent fused Claude Code's name, version, and role into one string
(e.g. claude-code_2-1-233_agent), fragmenting it into ~96 distinct values in
analytics and making a simple "how many Claude Code sessions" query
impossible. Replace the hand-ported detect-agent signature table with the
upstream github.com/vercel/detect-agent module (13 -> 20 harnesses, also
fixes a live OpenCode detection bug), and split any version-shaped suffix
off the detected name into a new persona.agent_version field, canonicalising
the remaining name fragment against the module's own vocabulary rather than
a hand-written alias list. persona.agent now always reports "claude_code"
for Claude Code regardless of detection path.

Needs OSS/legal sign-off before merge: the upstream module ships no LICENSE
file (see PR description).
@snyk-io

snyk-io Bot commented Aug 20, 2026

Copy link
Copy Markdown

Snyk checks have passed. No issues have been found so far.

Status Scan Engine Critical High Medium Low Total (0)
Open Source Security 0 0 0 0 0 issues
Licenses 0 0 0 0 0 issues
Code Security 0 0 0 0 0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

@github-actions

github-actions Bot commented Aug 20, 2026

Copy link
Copy Markdown
Contributor
Warnings
⚠️ There are multiple commits on your branch, please squash them locally before merging!
⚠️

"Merge remote-tracking branch 'origin/fix/CLI-1771-harness-name-version-split' into fix/CLI-1771-harness-name-version-split" is too long. Keep the first line of your commit message under 72 characters.

⚠️

"Merge remote-tracking branch 'fork/fix/CLI-1771-harness-name-version-split' into fix/CLI-1771-harness-name-version-split" is too long. Keep the first line of your commit message under 72 characters.

⚠️

"Merge branch 'fix/CLI-1771-harness-name-version-split' of github.com:snyk/cli into fix/CLI-1771-harness-name-version-split" is too long. Keep the first line of your commit message under 72 characters.

Generated by 🚫 dangerJS against df8c6e9

@snyk-pr-review-bot

This comment has been minimized.

@snyk-pr-review-bot

This comment has been minimized.

Comment thread cliv2/internal/persona/agent/agent_test.go Outdated
go-licenses save hard-fails on github.com/vercel/detect-agent because
its repo has no LICENSE/NOTICE/COPYING file, breaking make configure
and CI. Ignore it from go-licenses' auto-detection and supply the
Apache-2.0 text directly, matching the license package.json declares.

Ref: CLI-1771
…snyk/cli into fix/CLI-1771-harness-name-version-split
@snyk-pr-review-bot

This comment has been minimized.

Comment thread cliv2/internal/persona/agent/agent.go
Comment thread cliv2/internal/persona/agent/agent.go Outdated
Comment thread cliv2/internal/persona/persona_test.go

@robertolopezlopez robertolopezlopez left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Last comment: probably, those posible corner cases in agent.go make agent_test.go really necessary after the fix

detect-agent's README recommends `name@version` for custom AI_AGENT
declarations; versionSuffix only handled '_' and '/'. Also stop
TestReport_Agent from leaking ambient agent-signature env vars
(AI_AGENT, CLAUDECODE, etc.) from the shell running `go test` into
subtests that don't declare them.
…split' into fix/CLI-1771-harness-name-version-split
@snyk-pr-review-bot

This comment has been minimized.

@snyk-pr-review-bot

This comment has been minimized.

Comment thread cliv2/internal/persona/agent/vercel_contract_test.go
Comment thread cliv2/internal/persona/agent/agent.go Outdated
Same ambient-env leak as persona_test.go's TestReport_Agent, in the
two subtests of TestVercelDetectDoesNotNormalizeExplicitAIAgent that
call detectagent.Detect() directly. Also switch the isolateEnv
restore loop to t.Setenv, which golangci-lint's usetesting linter
requires over raw os.Setenv (and fixes an unchecked error to boot).
…n-split' into fix/CLI-1771-harness-name-version-split
@snyk-pr-review-bot

This comment has been minimized.

versionSuffix required 2+ numeric components to avoid matching a name
that merely ends in a digit, but that guard doesn't apply to '@': the
separator alone is the version signal per detect-agent's README,
whose own example ("devin@1") the old regex rejected. Split '@' into
its own atVersionSuffix pattern with a 1-component minimum; '_'/'/'
keep the 2-component guard since they can end an ordinary name.
@snyk-pr-review-bot

This comment has been minimized.

Two separate compiled patterns plus a match-fallback dance was more
machinery than the difference warranted. One regex now expresses both
cases via alternation with two version groups (version1 for '_'/'/',
version2 for '@'); SplitVersion just picks whichever fired. Behaviour
unchanged, same test suite passes.
@snyk-pr-review-bot

This comment has been minimized.

Comment thread cliv2/internal/persona/agent/agent.go Outdated
t.Setenv registers its own unset-cleanup; calling it from inside
isolateEnv's t.Cleanup undid the restore right after performing it,
leaving the caller's real environment empty instead of restored.
@snyk-pr-review-bot

This comment has been minimized.

'@' is never part of a Harness name (detect-agent's own documented
convention), so it's unambiguous even when the fragment in front of
it isn't a known harness — unlike '_'/'/', which are ordinary
characters a real name can end with. SplitVersion("custom-agent@2.0")
now returns ("custom-agent", "2.0") instead of leaving it unsplit.
@snyk-pr-review-bot

This comment has been minimized.

@nick-y-snyk
nick-y-snyk enabled auto-merge August 21, 2026 10:59
@snyk-pr-review-bot

Copy link
Copy Markdown

PR Reviewer Guide 🔍

🧪 PR contains tests
🔒 No security concerns identified
⚡ No major issues detected
📚 Repository Context Analyzed

This review considered 22 relevant code sections from 10 files (average relevance: 0.90)

🤖 Repository instructions applied (from AGENTS.md)

@nick-y-snyk
nick-y-snyk merged commit 27edd2c into main Aug 21, 2026
10 checks passed
@nick-y-snyk
nick-y-snyk deleted the fix/CLI-1771-harness-name-version-split branch August 21, 2026 13:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants